Checkout broke on the biggest sale weekend of the year

OXBOW sells about 40% of its year in nine days. The engineering team is nine people, and on the Saturday of the fall sale it is two.

The card vault had been upgraded on the Thursday. It passed staging, it passed a full checkout in production, and it quietly broke wallet payments — the ones where the customer never types a number — because the new client sent an idempotency key the vendor's older endpoint rejected after an eight-second wait.

Seventeen minutes

The alert rule was written for exactly this: more than fifty events in five minutes on the checkout project pages whoever is on call. It fired at 9:14 AM ET.

• The issue was already grouped — 612 events, one fingerprint, one route.
• The trace showed 8,000 ms sitting in a single outbound call, with the queries around it healthy.
• Suspect commits named the vault upgrade and the two people who reviewed it.

The rollback shipped at 9:31 AM ET. Nobody opened a log file.

What it saved

OXBOW puts the recovered revenue at $41,200, which is the difference between that Saturday's wallet conversion and the four Saturdays before it. The error rate across the whole nine days came in at 0.02%, down from 1.4% the year before.

The less countable part: the team now ships on Thursdays again. For two years they had a rule against deploying inside a sale week, which mostly meant a queue of forty changes going out together on the Monday after — the riskiest deploy of the quarter, made deliberately.